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CALIBRATION SYSTEM FOR VISION-BASED 
AUTOMATIC WRITING IMPLEMENT 

BACKGROUND 

1. Field of the Invention 

Embodiments described herein are directed to an automatically controlled writing 
implement with a digital camera, which draws or writes on a substrate and recognizes what a 
user draws or writes on that same substrate. 

2. Related Art 

Both interactive systems and non-interactive systems that use controllers to recognize a 
target are known in the art. For example, U.S. Patent No. 5,901 ,978, entitled "Method and 
Apparatus for Detecting the Presence of a Child Seat " teaches a method and system for 
detecting the presence of a child seat, more particularly to a method and apparatus for detecting 
the presence of a child seat in a rear-facing position, on a seat in which information about its 
contents is obtained and a signal is generated based on any contents of the seat. The analysis of 
the signal is preferably by pattern recognition techniques that can recognize and thus identify the 
contents of the seat. 

U.S. Patent No. 5,919,045, entitled "Interactive Race Car Simulator System," discloses 
an interactive vehicle simulator system that receives input signals indicative of the actions of a 
user who is located in a driver module. The simulator system also displays images and moves 
the driver module in accordance with the input signals. 

U.S. Patent No. 6,026,798, entitled "Professional Batting Training Machine," discloses a 
baseball pitching machine employing a counter-rotating wheel type baseball launch subsystem 
that pitches a series of baseballs as well as a computer controlled system for selecting the type 
and percentage of pitches, pitcher and batter characteristics, strike zone areas, and other 
parameters to provide a meaningful batting training session. 

U.S. Patent No. 6,188,484, entitled "Method and Apparatus for Measuring Angular 
Displacement of an Actuator Arm Relative to a Reference Position," further teaches an apparatus 
for measuring directly an angular displacement of an actuator arm relative to a reference 
position, wherein the actuator is rotatable about a fixed axis. 

The above-referenced examples are useful in their particular fields, such as for child 
safety, recreation, improvement of sports skills, and the like. The technology of recognizing a 
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target may also be employed in interactive games. Using a digital video camera for the purpose 
of detecting substrate movement, an automatically controlled pen may draw or write on a 
substrate and recognize what a user draws or writes on the same substrate. Problems may arise 
since the user may move the substrate while drawing, thereby confusing the drawing and 
recognition processes. To alleviate this problem, a controller draws a specialized asymmetrical 
target on the substrate initially. Before each drawing, the controller uses the camera to find and 
recognize the target and determine the orientation as well as the position of the substrate. This 
allows for interactive games such as tic-tac-toe, hangman, and the like to be played. 
BRIEF DESCRIPTION OF THE DRAWINGS 

In order that all of the structural and functional features for attaining the objects of the 
Calibration System for Vision-Based Automatic Writing Implement may be readily understood, 
a detailed description of embodiments of the invention will be made with reference to the 
accompanying drawings, wherein like numerals designate corresponding parts in the several 
figures. 

FIG. 1 is a flow chart illustrating the steps associated with a system for detecting 

substrate movement. 

FIG. 2a depicts a specialized target in the form of a circle. 
FIG. 2b depicts a specialized target in the form of a square. 

FIG. 3 depicts a drawing of a specialized target on a blank substrate with the use of an 
automatically controlled writing implement. 

FIG. 4 depicts a digital video camera observing the target drawn on the substrate. 

FIG. 5 depicts the drawing on the substrate by the automatically controlled writing 
implement based on the position and orientation of the substrate. 

FIG. 6 depicts a computer interfaced to the base of an articulated robot arm. 
TtF.TATT.ED DESCRIPTION 

The following paragraphs describe a calibration system for a vision-based writing 
implement according to an embodiment of the present invention. According to one embodiment 
of the present invention, as illustrated in FIG. 3, an articulated robot arm 320, secured to a base 
325, has software-controlled mechanical actuators and sensors 335 and a digital video camera 
330 appended to it. The digital video camera 330 may also be mounted in some other way, such 
as on a tower mounted to the base 325 of the articulated robot arm 320. An automatically 
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controlled writing implement 315 draws or writes on a substrate 310. The substrate 310 may be 
a sheet of paper, a blackboard, a whiteboard, or the like. Through the use of the digital video 
camera 330, the system recognizes what a user draws or writes on the same substrate 310. It is 
assumed that the substrate 310 does not move while the automatic vrating implement 315 is 
5 drawing. The user, of course, may move the substrate 310 while drawing or writing, thereby 
confusing the drawing and recognition processes. 

FIG. 1, together with FIG. 3, illustrates the overall operation of a calibration system for a 
vision-based automatic writing implement according to an embodiment of the present invention. 
As an example, the system may engage in an interactive game of tic-tac-toe with a user. The 

10 game may be pre-loaded into the system through an applicable CD ROM disk drive or floppy 
disk drive 340. The game may further be built into the system. Moreover, the game may be 
located on a separate computer 610, as shown in FIG. 6. In such a case, the articulated robot arm 
320 would provide sensor data to the computer 610 and the game itself would run on the 

# computer 610. 

jlj The user places a blank substrate 310 in proximity to the articulated robot arm 320. A 

base 325 supports the articulated robot arm 320. The articulated robot arm 320 is capable of 

ft] picking up and holding small objects, namely an automatically controlled writing implement 
315. The writing implement 315 may be a pen, pencil, marker, crayon, piece of chalk, or the 

|r^^ like. The arm 320 is of such a size and type to be suitable for a child of approximately eight 

I j| years of age. Appended to the articulated robot arm 320, in this embodiment, is a digital video 

|il camera 330. The digital video camera 330 sends video to the controller of the computer 610. 

p;^; The computer 610 is connected to the apparatus via an appropriate serial or parallel interface 
615. Wireless connection is also possible. The controller may further be built into the apparatus 
such as in the articulated robot arm 320 or in the base 325. Moreover, parts of the controller may 

25 be located within the apparatus such as in the articulated robot arm 320 or in the base 325, and 
other parts of the controller may be located in a separate computer 610. The digital video camera 
330 is similar in operation to a personal computer camera sold by Intel Corporation such as the 
Intel® Easy PC Camera, Intel® Deluxe PC Camera, Intel® Pro PC Camera, or Intel® Pocket PC 
Camera. 

30 The system, using the digital video camera 330, recognizes that the substrate 310 is 

blank, as represented by step 110. The digital video camera 330 acts as an image sensor for the 
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system. Next, the automatically controlled writing implement 315 is employed by the system to 
draw a specialized target 210b on the substrate 310, as shown in step 115 and illustrated in FIG. 
3. FIG. 2a depicts a specialized target 210a in the form of a circle. FIG. 2b depicts another 
specialized target 210b in the form of a square. The specialized target 210 may be of a wide 
5 variety of shapes. At this stage, the writing implement 315 may also draw the tic-tac-toe board 
410, as shown in FIG. 4, as well as make the first play by placing either an "X" or an in one 
of the nine squares. The system observes and remembers the relative positions and orientations 
of the specialized target 210b and the drawing, here the tic-tac-toe board 410. 

The user is then allowed to make a move on the game board 410. While the user is 
10 marking the substrate 310, the digital video camera 330 is placed in such a position to observe 
the substrate 310, as illustrated by step 120. In each video frame, as provided by the digital 
video camera 330 to the controller of the computer 610, the system searches for the specialized 
target 210b, as shown by step 125. The result of this search includes both the location of the 
specialized target 210b in the video frame and its orientation. That is, the controller uses the 
digital video camera 330 to find and recognize the specialized target 210b and to determine 
whether the substrate 310 has moved, as shown by step 130. The target 210b is asymmetrical, 
thereby allowing for the determination of the orientation and location of the substrate 310. 
Software analysis by way of pattern matching occurs in determining whether the substrate 310 
has rotated. The controller can then orient its drawing and any recognition algorithms to the new 
location and orientation of the substrate 310, if the substrate 310 has indeed moved. The 
necessary software may be loaded into the apparatus itself. It may fiarther be installed on the 
computer 610 and downloaded. 

If the system finds the specialized target 210b, it then locates the drawing, here the tic- 
tac-toe board 410, in the video frame relative to the specialized target 210b, as illustrated in step 
25 135. By examining the tic-tac-toe board 410 in the video frame, the system determines whether 
the user marked the substrate 310, as represented by step 140. The system repeats steps 125, 
130, 135, and 140 until the system determines that the user has marked the substrate 310, i.e. 
made a move on the tic-tac-toe board 410. 

At this time, the interactive system proceeds to make its own move, using the position of 
30 the specialized target 210b to find the location of the substrate 310 where it should draw. This 
process is shown as step 145. The calibrations system then determines whether the game is 
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completed, as illustrated in step 150. If the game has not reached completion, the system retiims 
to step 120 of allowing the user to make another marking on the substrate 310. The entire 
process continues until the game is completed. If the game has reached completion, the system 
returns to step 110. The same process would take place for any other kind of interaction that the 
5 system might perform. As suggested above, hangman is another viable option whereby the 
system would draw the blanks, and the user would fill in the blanks with letters of the alphabet. 
The system, which is pre-programmed with the final pattern, would also draw an limb of a 
person's body each time that the user incorrectiy placed a letter in a blank. It should be 
appreciated that tic-tac-toe and hangman are not the only interactive games that a user may play 
10 with the system. They are named merely for illustrative purposes. 

FIG. 5 illustrates the drawing on the substrate 310, by the automatically controlled 
writing implement 315 based on the position and orientation of the observed specialized target 
tsj 210b. In this drawing configuration, the controller cannot observe the speciaUzed target 210b. 
B The controller has, however, already determined where it should draw by observing tiie position 
U of the target 210b. As ilhastrated here, the system has already drawn the tic-tac-toe board 410, 
% made a move by placing an "X" in the middle square, allowed the user to mark tiie substi-ate 310, 
' H as represented by the dotted "O", observed the substirate 310, searched for the specialized target 
= ' 210b in a plurality of video fi-ames, determined whether the substrate 310 moved, located tiie 
H drawing in the video frame relative to tiie specialized target 210b, and examined the drawing and 

M determined that the user in fact marked tiie substrate 310 with an "O." As such, FIG. 5 

J*? 

corresponds to step 145 in FIG. 1 , whereby the system makes a subsequent drawing, here the 
.M- -- second "X" by using the position of tiie specialized target 210b to find tiie location of the 
substirate 310 where it should draw. 

While die above description refers to particular embodiments of the present invention, it 
25 will be understood to those of ordinary skill in the art tiiat modifications may be made without 
departing from the spirit thereof. The accompanying claims are intended to cover any such 
modifications as would fall vdthin the true scope and spirit of the present invention. 

The presentiy disclosed embodiments are therefore to be considered in all respects as 
illustirative and not restrictive; the scope of the invention being indicated by the appended claims, 
30 rather than tiie foregoing description. All changes tiiat come within tiie meaning and range of 
equivalency of the claims are therefore intended to be embraces therein. 
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